Skip to content

453 csv upload in new design#88

Merged
foobacca merged 32 commits intomainfrom
453-csv-upload
Mar 27, 2026
Merged

453 csv upload in new design#88
foobacca merged 32 commits intomainfrom
453-csv-upload

Conversation

@gazdagergo
Copy link
Collaborator

@gazdagergo gazdagergo commented Mar 22, 2026

ticket: 453

Copy link
Contributor

@foobacca foobacca left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's WIP but I thought I'd have a quick look.

@foobacca foobacca marked this pull request as ready for review March 24, 2026 11:07
gazdagergo and others added 13 commits March 24, 2026 13:27
Add admin-only developer tool page at /backoffice/dev/service-docs that
provides interactive documentation for CSV upload service functions.
Disabled in production for security.

Features:
- Documents import_respondents_from_csv, import_targets_from_csv,
  get_or_create_csv_config, and update_csv_config services
- Shows code references, parameters, return types, and error cases
- Interactive "Try It" sections to execute services directly
- Sample data loading for testing
- Tab navigation with URL query parameter sync
- Adds x-cloak CSS rule for Alpine.js CSP build compatibility

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace nested x-model paths with flat properties (CSP build limitation)
- Update execute/reset/sample loader methods to use flat properties
- Add X-CSRFToken header to fetch request for CSRF protection
- Add x-cloak to loading spinners to prevent flash

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create /backoffice/dev/patterns route for interactive pattern docs
- Add patterns.html with dropdown pattern examples (URL select, inline state)
- Include live examples, code snippets, CSP notes, and implementation index
- Update AGENTS.md with Alpine.js CSP constraints reference

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use & separator when URL already has query params, otherwise use ?
- Fixes URL like ?tab=dropdown?demo=option (invalid) becoming ?tab=dropdown&demo=option
- Add server-side selected attributes to patterns page dropdown

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create /backoffice/dev route with cards linking to dev tools
- Links to Service Docs, Frontend Patterns, and Component Showcase
- Admin-only, disabled in production
- Update breadcrumbs in service_docs and patterns to link to dev dashboard

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
UI Component:
- Add file_input macro to components/input.html
- Supports label, hint, error, accept, required, disabled
- Tailwind styling with file: pseudo-class for button

Pattern Documentation:
- Add File Upload tab to frontend patterns page
- Document architecture (UI -> Flask route -> service layer)
- Live demo with client-side file validation and preview
- Template and Flask route code examples
- Implementation index linking to existing code

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add file_input to input component showcase
- Show basic, required with hint, error, and disabled states
- Update description and code example to include file_input

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The DjHTML linter reformatted the usage examples in a way that
broke Jinja comment parsing, causing the examples to be executed
as actual code. Simplified to plain text examples matching the
pattern used by other macros.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add shared assembly_tabs macro for consistent tab rendering
- Create Targets page showing gsheet config info or CSV placeholder
- Create Respondents page showing gsheet config info or CSV placeholder
- Add routes: /assembly/<id>/targets and /assembly/<id>/respondents
- Update all assembly templates to use shared tabs macro
- Tabs are always visible but disabled until data source is configured
- For gsheet source: tabs show info about configured sheet tab names
- Add BDD tests for tab visibility and enabled/disabled states
- Add docs/agent/453-csv-upload.md for feature documentation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add service functions: get_csv_upload_status, delete_targets_for_assembly,
  delete_respondents_for_assembly
- Add routes for uploading/deleting targets and respondents CSV files
- Update assembly_data.html with CSV upload panel showing Target and People
  upload sections with proper state management
- Lock data source selector when CSV data is uploaded
- Enable Targets/Respondents tabs based on uploaded data
- Add BDD tests for CSV upload scenarios
- Include sample CSV files for testing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Selection tab to /backoffice/dev/service-docs with documentation for:
  - check_db_selection_data()
  - start_db_select_task()
  - get_selection_run_status()
  - generate_selection_csvs()
  - cancel_task()
  - check_and_update_task_health()
  - get_respondent_attribute_columns()
  - translate_run_report_to_html()
- Update 453-csv-upload.md with sortition service layer functions
- Add "selection" to valid tabs in service_docs route

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Revert accidental downgrade and update to latest version.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Selection tab is now disabled when no data source is configured.
Updated test scenarios to include gsheet configuration where needed.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
foobacca and others added 7 commits March 24, 2026 14:44
By specifying Callable args and return type
Replace one-at-a-time session.delete() loops with single DELETE WHERE
statements in both TargetCategory and Respondent repositories.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
rather than a dict - makes the code easier to read and reason about
… service functions

Cover delete_targets_for_assembly and delete_respondents_for_assembly
with service-level tests (permissions, not-found, isolation, happy path)
and add repository-level tests for respondent bulk deletion.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove the restriction that required targets to be uploaded before
people. Both CSV files can now be uploaded in any order.

Update BDD test to match the new behavior.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@gazdagergo gazdagergo changed the title WIP: 453 csv upload in new design 453 csv upload in new design Mar 25, 2026
@gazdagergo gazdagergo requested a review from foobacca March 25, 2026 12:17
foobacca and others added 5 commits March 25, 2026 12:27
Use COVERAGE_PROCESS_START mechanism instead of wrapping commands with
`coverage run`. Coverage's installed .pth file auto-instruments Python
subprocesses when the env var is set, and pytest-cov automatically
combines the resulting data files during its finalization.

- Replace _build_coverage_command() with _add_coverage_env() that sets
  COVERAGE_PROCESS_START and COVERAGE_FILE per subprocess
- Use --pool=solo for celery when collecting coverage to avoid prefork
  workers fighting over the data file
- Use start_new_session + os.killpg for clean process group termination
- Remove manual coverage combine step from justfile (pytest-cov handles it)

Also change the justfile to use spaces instead of tabs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Documents blueprint and service layer architecture with visual diagrams
showing dependencies, route mappings, and recommendations for potential
code organization improvements.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Move /backoffice/dev/* routes to a separate dev.py blueprint for
cleaner separation of developer tools from production code.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Allow users to specify which column contains unique identifiers when
uploading respondents CSV. If left blank, the first column is used.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
foobacca and others added 4 commits March 26, 2026 22:26
* main:
  regenerate translations following justfile fix for pybabel args
  fix linting with auto reformat
  Apply suggested fix to backend/tests/fakes.py from Copilot Autofix
  Apply suggested fix to backend/tests/fakes.py from Copilot Autofix
  Apply suggested fix to backend/justfile from Copilot Autofix
  Apply suggested fix to backend/justfile from Copilot Autofix
  Apply suggested fix to backend/justfile from Copilot Autofix
  trigger flask reload (if running) when rebuilding css
  upgrade requests for security fix (and other pkgs)
  Have .well-known/change-password always go the change password form
  test: add contract tests for all repositories, fix 2 fake bugs
  test: add contract test pattern for PasswordResetTokenRepository
  edits to well known files
  feat: add well-known URL endpoints for robots.txt, security.txt, and change-password
  build(deps): Bump picomatch in /backend
  fix: auto-confirm email for users created via CLI commands
  add agent plan for invite email improvements
  feat: persist invite email address on UserInvite domain model
  Bump sass from 1.97.3 to 1.98.0 in /backend
Instead of checking is_production() in each route handler, only
register the dev blueprint when not in production. This means the
routes don't exist at all in production.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Visual overview of E2E and BDD test coverage for backoffice routes
and services, with clear indicators for covered/uncovered code and
test types (happy path, validation, permission, etc.).

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The new dev blueprint is not even registered with the flask app when the
config is production (which includes the public demo site) so we don't
care about coverage of that.

We now track coverage of BDD tests, so we don't need to exclude them any
more.
@codecov
Copy link

codecov bot commented Mar 27, 2026

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment

Thanks for integrating Codecov - We've got you covered ☂️

foobacca and others added 2 commits March 27, 2026 12:14
Add E2E tests for previously uncovered happy paths:
- upload_targets_csv: successful targets CSV upload
- delete_targets: successful targets deletion
- delete_respondents: successful respondents deletion
- view_assembly_targets: targets page for CSV data source
- view_assembly_respondents: respondents page for CSV data source

Add BDD scenario for update_number_to_select from selection page.

Add fixtures for assembly_with_targets and assembly_with_respondents.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@foobacca foobacca merged commit 514aa92 into main Mar 27, 2026
15 of 17 checks passed
@foobacca foobacca deleted the 453-csv-upload branch March 27, 2026 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants